(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(11) 



EP1 220 157 A2 



(12) 



EUROPEAN PATENT APPLICATION 



(4J) 


Date of publication: 


(51) lntCl7: G 061 9/00 




HT onno RuiiAtin onno/07 
uo.u/.^uu^ Duiieiin ^uu^/^/ 


(21) 


Application number: 01310865.9 




(22) 


Date of filing: 24.12.2001 




(84) 


Designated Contracting States: 


(72) Inventor: Laaksonen, MIka H. 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


90500 Oulo (Fl) 




MCNLPTSETR 






Designated Extension States: 


(74) Representative: Ruuskanen, Juha-Pekka et al 




AL LT LV MK RO SI 


Page White & Farrer, 






54 Doughty Street 


(30) 


Priority: 29.12.2000 GB 0031825 


London WC1N2LS (GB) 


(71) 


Applicant: Nokia Corporation 






02150 Espoo (Fl) 





(54) Image processing 



(57) Tlie present invention relates to image 
processing. In tlie disclosed method an image area is 
divided into a plurality of image data blocl<s. Each of the 
image data blocks is compressed and stored in a data 
storage means. Before an image is displayed based on 
the stored data, at least one of the image data bloclcs is 
selectively fetched from the data storage means. After 



fetching said selected at least one image data block said 
at least one image data block is decompressed and the 
image is displayed based on image data included in the 
fetched image data blocks. In the invention modification 
to the boundaries of the image area are made by means 
of the selective fetching without modifying the image da- 
ta stored in the data storage means. 
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Description 

[0001] The present invention relates to image 
processing and in particular, but not exclusively, to dis- 
playing innages based on innage data stored in innage 
data storage nneans. 

[0002] An image can be represented by means of an 
appropriate image representation device. For example, 
an image can be displayed by a display device compris- 
ing a display screen or other means for presenting an 
image based on image data. The image data can be cre- 
ated, for example, by means of a digital camera. When 
a photograph is taken by the digital camera, the digital 
camera creates an image data file that may be stored 
in a memory means of the camera. The image data may 
also or alternatively be stored in a removable data car- 
rier, such as in a tape, CD (compact disc) or a memory 
card. Digital cameras are well known in the art, and will 
thus not be explained in more detail herein. 
[0003] Typically the image or audio data is stored in 
a file, record or similar data unit. It is possible to com- 
press the data stored in the data unit. The compressed 
data requires less storage capacity when stored in an 
appropriate data storage means. A reason for the com- 
pression is that the digital cameras or other portable de- 
vices are typically provided only with a limited storage 
capacity. Furthermore, if the data unit is to be transport- 
ed between two or more devices, compressed data files 
require less transportation capacity from the communi- 
cation media between the devices. 
[0004] An example of compressed data units is a 
compressed JPEG file for storing image data. The term 
'JPEG' stands for Joint Photographic Experts Group 
which Is formed by experts nominated by standards 
bodies and major companies. A standard by the JPEG 
is IS 10918-1 (ITU-T T.81), which is the first of a multi- 
part set of standards for still image compression. JPEG 
is based on the features of this standard, in association 
with a JFIF file fonnat by C-Cube Microsystems. 
[0005] Digital cameras typically compress images 
with a so called lossy compress algorithm before storing 
these to a memory. The above referenced JPEG sup- 
ports the lossy compression. By means of the lossy 
compression the size of the image files can be made 
small enough so that it is possible to store the image 
data in limited storage capacity of the camera. A disad- 
vantage of the lossy compression technique is that it los- 
es information during the compressing operations. 
[0006] One important feature of the digital photogra- 
phy is that it is possible to adjust and edit the images 
aften/vards. The changes can be made by anybody. That 
is, the creator (i.e. the photographer) or the viewer or 
any other person in a chain between the creator and the 
viewer may edit or otherwise alter the image. 
[0007] A possible editing task is the so called image 
cropping. In the image cropping unnecessary or other- 
wise unwanted areas of the image are cut out from the 
boundary areas of the image. In that way image size 



resolution, and thus size of the image file can be re- 
duced. 

[0008] An unwanted side effect of the editing process- 
es is that the quality of the image may be reduced. For 

5 example, if an image is cropped and stored successive 
times the quality of image may be reduced in each ed- 
iting cycle comprising decompression - compression 
and storing. This is caused by the possibility of loosing 
information during the image compression stages, es- 

10 pecially if a lossy compression algorithm is used. 

[0009] Thus the quality of the image may be reduced 
every time the image is fetched from, decompressed, 
compressed and stored again in the image data file. 
Since this cycle may happen every time the image data 

15 is processed, the quality of the image may get progres- 
sively worse each time an image is subjected to modi- 
fications. 

[0010] The above discusses storing of the images in 
a lossy compression format. Another possibility is to 

20 store images by so called lossless compression for- 
mats. J PEG2000 is an example of a compression stand- 
ard supporting both lossy and lossless compression. 
However, the lossless formats require more memory ca- 
pacity for the image data storing than the lossy formats. 

25 Therefore the lossless formats may not be suitable for 
substantially small portable devices, such as for exam- 
ple digital cameras or display devices. 
[0011] Digital imaging is gaining more popularity. An 
especially rapidly growing area of digital image handling 

30 is believed to be transportation of digital image data over 
data networks. At least the transmitting device or the re- 
ceiving device may be a portable user equipment or oth- 
er mobile station capable of communication over a wire- 
less interface with the data network apparatus. As the 

35 size and/orweight is typically an important feature of the 
portable devices, the above discussed limitations in size 
and memory capacity need to be taken into considera- 
tion when designing such devices. 
[0012] The end users of the image capturing or dis- 

40 playing devices may also wish to be more readily able 
to define the borderlines of an image. To enable this it 
is likely that even the user equipment will be provided 
with at least some level of borderline adjustment facility. 
The inventor believes also that at least in the first wire- 

^5 less image data transmission devices the image com- 
pression methods are likely to be based on lossy com- 
pression techniques. This is believed to be so because 
of the pixel count in the image and demands on storing 
capacity growfasterthan technologies that may be used 

50 to increase the storage capacity. However, the deterio- 
rating quality of the images may become a problem also 
with devices that are based on the lossless techniques. 
[0013] Embodiments of the present invention aim to 
address one or several of the above problems. 

55 [0014] According to one aspect of the present inven- 
tion, there is provided a method of storing image data 
and displaying images based on the stored image data, 
comprising: dividing an image into a plurality of image 
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data blocks; compressing each of the image data blocks 
separately; storing the compressed image blocks in a 
data storage means; selecting at least one of the image 
data blocks to be fetched from the data storage means; 
fetching said selected at least one image data block 
from the data storage means; decompressing said at 
least one fetched image data block; and displaying an 
image based on image data included in the decom- 
pressed at least one image data block. 
[001 5] According to another aspect of the present in- 
vention there is provided a display device for displaying 
images based on image data, wherein image data as- 
sociated with an image has been divided into a plurality 
of image data blocks before being stored in a com- 
pressed format in an image data storage means, the dis- 
play device comprising: selection means for selecting at 
least one of the image data blocks to be fetched from 
the image data storage means; fetching means for 
fetching said selected at least one image data block 
from the data storage means; decompressing means for 
decompressing said at least one fetched image data 
block; and display means for displaying an image based 
on image data included in the fetched at least one image 
data block. 

[0016] The arrangement is preferably such that the 
changes in the image boundaries to be displayed do not 
affect the image data stored in the data unit. 
[001 7] The selection means may be adapted to adjust 
the boundary of the image to be displayed by selecting 
appropriate image data blocks. The selection means 
may be operated based on a preview. 
[001 8] Each of the image data blocks may be provid- 
ed with an identifier. 

[0019] The compressed image data blocks may be 
stored in an image data field of an image data storage 

unit and information associated with the selection of im- 
age data blocks is stored In another field of the image 
data storage unit. 

[0020] The selected at least one block may be copied 
to a new image data file from the compressed image 
data blocks. 

[0021] The image may be displayed on a display 
screen of a portable display device. The image data may 
be transmitted to the display device over a wireless in- 
terface. 

[0022] The embodiments may enable edition and/or 
adjustment of image boundaries without any substantial 
reduction in the quality of the image data. The embodi- 
ments may provide an easy way to undo changes in the 
image area and return to the original image without loss- 
es in quality or information content. An image can be 
adjusted in several different ways without affecting the 
original image information. For example, the embodi- 
ments may enable storage of two different image areas 
and selection one of these based on the conditions or 
situations in which the image is displayed. 
[0023] For better understanding of the present inven- 
tion, and ways to put the invention into effect, reference 



will now be made by way of example to the accompa- 
nying drawings in which: 

Figure 1 shows an image presentation means in 
5 which embodiments of the present invention may 
be utilised; 

Figures 2a and 2b illustrate an embodiment of the 
invention; 

Figure 3 illustrates a further embodiment; 
10 Figure 4 is a flowchart illustrating the operation of 
an embodiment of the present invention; and 
Figures 5a and 5b illustrate a further embodiment 
of the present invention. 

15 [0024] Reference is made to Figure 1 which shows a 

portable user equipment 1 comprising a display means 
2. More particularly, the user equipment 1 comprises a 
mobile station capable of communication over a data 
communication network via a wireless interface be- 
20 tween antenna means 7 and the network. Elements of 
the data communication network are not shown for clar- 
ity reasons. 

[0025] The operation of the representation device 1 
and/or the display screen 2 may be controlled by a con- 
25 trol processors. The device 1 is also preferably provided 

with data storage means 4. An operational connection 
6 is provided between the data storage means 4 and the 
control processor 5. 

[0026] The display means 2 is adapted to present im- 
30 ages based on image data. The image data may be 
stored in the storage 4. The image data may be proc- 
essed by the processor 6 before displaying the image 
to the user. The processing may be accomplished based 
on a control software run on the processor. 
35 [0027] The device 1 may comprise one or more con- 
trol buttons 3. The control buttons may be used to con- 
trol the operation of the device and also to edit the image 
before and during the display thereof. 
[0028] The schematic exemplifying image displayed 
40 by the display screen 2 illustrates two boxes and a ball. 
The display means may comprise any appropriate dis- 
play arrangement, such as a liquid crystal display (LCD). 
It shall be appreciated that the display means of Figure 
1 is only an example, and that the display means does 
45 not form an essential part of the invention, The display 
means may alternatively comprise e.g. a printer device 
producing a reprint of the image based on image data 
processed by the user device 1 . Since various alterna- 
tives for the user devices and possible display means 
50 as such are well known by the skilled person, these will 
not be described in more detail herein. 
[0029] The following will discuss in more detail exem- 
plifying embodiments of the invention. In these exam- 
ples a JPEG file format is used for storing and/or trans- 
55 porting image data. JPEG file format is a well known for- 
mat in the art of storing images using lossy compres- 
sion, and the principles thereof will thus not be explained 
in more detail. 
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[0030] The embodiments relate to prevention of qual- 
ity losses when adjusting image boundaries, i.e. during 
the so called image cropping. The embodiment is de- 
scribed with reference to Figures 2a and 2b and the flow- 
chart of Figure 4. 

[0031] The original image 30 has been divided into 
smaller image data blocl<s 31 . The image data blocl<s 
30 may be, for example, 8 pixel * 8 pixel blocks. Lossy 
JPEG compression may be used for compressing each 
of the image data pixel blocks 31 one by one to com- 
pressed format. In the compressed format the blocks 31 
are preferably stored in the same order as they are in 
the original image 30. 

[0032] It is sometimes necessary to cut of the border 
areas of the image 30, i.e. to crop the image. For exam- 
ple, the user of the representation device may wish to 
adjust the boundaries of the image, e.g. so that the im- 
age fits the available display screen area of the display 
device. The cropped area of the image 30 of Figure 2a 
is shown to comprise the six blocks 31 within the rec- 
tangular cropping window 32. The "cropped" image is 
shown in Figure 2b. 

[0033] The selection of the image data blocks that are 
to be displayed may be based on a preview of the image. 
The preview is a known feature of the display devices. 
During the preview the user of the device may select the 
image area he/she wishes to be displayed. The cropping 
is then accomplished accordingly in response to the se- 
lection. The cropping can be performed under the con- 
trol of the processor 5 of the display device 1 . 
[0034] According to a possibility all image data blocks 
are decompressed before the preview. After the image 
area selection the decompressed data is simply deleted 
and the cropped image is based on those image data 
blocks that have fetched based on the selection. 
[0035] The display device may be provided with a 
control software that is adapted to select the appropriate 
blocks. For example, during the preview the user points 
the image area he/she wishes to be displayed. The con- 
trol software is aware of the division of the pixels into 
the blocks. Based on the information about the division 
it may suggest which blocks need to be selected in order 
to obtain an image that substantially corresponds the 
selected area, After the selection these blocks are 
fetched from the storage means and the selected image 
area is displayed on the display. 

[0036] Figures 3 illustrates an embodiment of the 
present invention. In here it is assumed that a 40x40 
(=5*8x5*8) pixels image has been created. It is pre- 
ferred to have an appropriate name for each of the 
blocks so that they may be identified later on. 
[0037] The identifier can be name a of the block or e. 
g. the location of the block in the image area. 
[0038] In Figure 3 alphabets from a to t have been 
designated to represent twenty 8x8 pixel blocks 31 in a 
JPEG image file. The twenty blocks (a to t) are in a com- 
pressed domain and thus the blocks cannot be dis- 
played before decompression thereof. When cropping 



the image, it is possible to decide that only the center or 
other predefined area of the image is to be displayed. 
In Figure 3 blocks 'ghiimn' of the image 30 are selected 
for the cropped image 32. This may be enough e.g. for 
5 displaying an image of a face. 

[0039] The cropping may be done without affecting 
the original JPEG image data on the image data file. The 
modifications to the boundaries of the image area are 
made by means of the selective fetching without medi- 
co fying the image data stored in the data storage means. 
Thus it is not necessary to perform the decompress and 
compress cycle for the original image data file to get the 
cropped image, Even if the entire image data file has 
been decompressed for the preview, it is not necessary 
15 to restore, but the decompressed preview data may be 
deleted. Thus it is possible to avoid quality losses that 
are caused by the decompression-compression cycle to 
the original image data. 

[0040] The six blocks 'ghiimn' may be selected and 
20 copied to a new image file. The cropped image may then 
be displayed based on the new file. The pixel blocks 
'ghiimn' may copied e.g. to a new JPEG file. The image 
data associated to image 32 of Figure 2b may be stored 
after viewing for later use or it may deleted. 
25 [0041] Figures 5a and 5b illustrate a further embodi- 
ment. In this embodiment the image data file 20 is pro- 
vided with a data field 21 and a comment field 22. The 
comment field may be used for storing free form infor- 
mation that associates with the border line adjustments. 
30 The data associated with the changes in the image area 
can be stored beforehand to the JPEG file by storing 
this data in the comment field. By means of this it may 
be possible to avoid manual selection and/or preview 
operations. 

35 [0042] More particularly, in Figure 5a the comment 
field 22 comprises selection information that is associ- 
ated with a certain type of display device (e.g. a certain 
display screen size or type). That is, the comment field 
indicates those blocks of image 31 of Figure 3 that 
40 should be cropped when the image is to be displayed 
with a specific type of a device. 
[0043] The comment field or other free format data 
field may also include initial settings forthe cropping pa- 
rameters that were set when the image data was first 
45 created as a basic setting to be used when displaying 
the image. These settings may always be used as a de- 
fault. 

[0044] It is also possible to store information in the 

comment field regarding parameters that are preferred 
50 by a user of a device. A user may also later on adjust 
the default or previous settings to fit better the actual 
use of an image. Figure 5b shows the situation in which 
the parameters in field 22 have been changed from the 
parameters shown in Figure 5a after the image data has 
55 been fetched and the viewer of the image has adjusted 
cropping parameters to suit better for his/hers use. 
[0045] The above described usage of the comment 
field of a JPEG file. This does not does not prevent view- 
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ing an image based on image data stored in tine JPEG 
file by means of programs and display applications 
which are not capable of understanding the information 
content in the comment field 22. The comment field 22 
is a part of the JPEG standard, and the JPEG compat- 
ible devices are typically adapted to ignore any informa- 
tion in the comment field they cannot interpret and/or 
utilise. There is no limitations for the content that may 
be stored in said comment field. The way how the infor- 
mation associated with the cropping of the image is 
stored in the separate data field may be standardised, 
at least to some level, so that different software appli- 
cations that write/read information in the separate data 
field may recognise the information associated with the 
amendments. 

[0046] The decompression - compression cycle and/ 
or decompression of all of the image data for the preview 
purposes can also be avoided by displaying only a rough 
or scaled version of the image at the preview stage. For 
example, a so called thumb nail version of the image 
may be provided in the Image data unit for the preview 
purposes. Some formats may also allow decompression 
of only a portion of the image data in each block for the 
preview such that only a rough image is displayed to the 
viewer. The viewer may then select the image area he/ 
she wants to view based on the main fomns and/or fea- 
tures of the image that are made visible to him. 
[0047] The embodiment of the present invention has 
been described in the context of a JPEG format. This 
invention is also applicable to any other compression 
and/or processing techniques. For example, it is possi- 
ble to store image data in a JPEG 2000 file or any other 
data format which supports division of the image area 
into a plurality of blocks. 

[0048] It shall be appreciated that whilst the Figure 1 
embodiment of the present invention was described in 
relation to a mobile station, embodiments of the present 
invention are applicable to any other suitable type of us- 
er equipment capable of presenting an image or audio 
presentation to the user thereof . These devices include, 
without limiting to these, digital cameras and recorders, 
various display devices provided with image editing fa- 
cility and so on. 

[0049] It is also noted herein that while the above de- 
scribes exemplifying embodiments of the invention, 
there are several variations and modifications which 
may be made to the disclosed solution without departing 
from the scope of the present invention as defined in the 
appended claims. 



compressing each of the image data blocks 
separately; 

storing the compressed image blocks in a data 
storage means; 

5 selecting at least one of the image data blocks 

to be fetched from the data storage means; 
fetching said selected at least one image data 
block from the data storage means; 
decompressing said at least one fetched image 

10 data block; and 

displaying an image based on image data in- 
cluded in the decompressed at least one image 
data block. 

15 2. A method as claimed in claim 1 , comprising adjust- 
ing the size of the image area to be displayed by 
means of said selection of at least one image data 
block. 

20 3. A method as claimed in claim 1 or 2, wherein the 

initial compressed data remains substantially un- 
changed during the display procedures. 

4. A method as claimed in any preceding claim, com- 
25 prising the steps of displaying a preview of the 

stored image and selecting the image data blocks 
based on said preview. 

5. A method as claimed in any preceding claim, where- 
to in the image data is decompressed before the se- 
lection of at least one image data block to be 
fetched. 

6. A method as claimed in any preceding claim, where- 
35 in the image data blocks are selected based on a 

suggestion by a selection software run on a control- 
ler of a display device. 

7. A method as claimed in any preceding claim, where- 
40 in each of the image data blocks is provided with an 

identifier and the selection and fetching of the 
blocks is based on said block identifiers. 

8. Amethod as claimed in any preceding claim, where- 
45 in the image data blocks are stored in the same or- 
der as they are positioned in the original image. 

9. Amethod as claimed in any preceding claim, where- 
in said compressed image data blocks are stored in 
an image data field of an image data storage unit 
and information associated with the selection of im- 
age data blocks is stored in another field of the im- 
age data storage unit, wherein at least one block is 
selected based on said infomnation. 

10. A method as claimed in claim 9, wherein said other 
field comprises a comment field. 



50 



Claims 

1. A method of storing image data and displaying im- 
ages based on the stored image data, comprising: 55 

dividing an image into a plurality of image data 
blocks; 
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1 1 . A method as claimed in claims 9 or 1 0, wherein the 
most recent information associated with the selec- 
tion of the image data blocks is stored in the said 
other field. 

12. A method as claimed In any preceding claim, where- 
in the selected at least one block is copied to a new 
image data file from the compressed image data 
blocks. 

13. A method as claimed in claim 12, wherein the at 
least one block is stored in an image data field of 
the image data file. 

1 4. A method as claimed in any preceding claim, where- 
in the image data is stored in a lossy format. 

15. A method as claimed in claim 14, wherein the image 
data storage means comprises a JPEG file or sim- 
ilar. 

16. A method as claimed in any preceding claim, where- 
in the image is displayed on a display screen of a 
portable display device. 

17. A method as claimed in claim 16, wherein the image 
data is transmitted to the display device over a wire- 
less interface. 

1 8. A display device for displaying images based on im- 
age data, wherein image data associated with an 
image has been divided into a plurality of image da- 
ta blocks before being stored in a compressed for- 
mat in an image data storage means, the display 
device comprising: 

selection means for selecting at least one of the 
image data blocks to be fetched from the image 
data storage means; 

fetching means for fetching said selected at 
least one image data block from the data stor- 
age means; 

decompressing means for decompressing said 

at least one fetched image data block; and 
display means for displaying an image based 
on image data included in the fetched at least 
one image data block. 

19. A display device as claimed in claim 1 8, wherein the 
selection means are adapted to adjust the boundary 
of the image to be displayed by means of selecting 
appropriate image data blocks. 

20. A display device as claimed in claim 1 8 or 1 9 being 
arranged to display a preview of the stored image, 
wherein the selection means are operated based 
on said preview. 



21. A display device as claimed in any of claims 18 to 

20, wherein the selection means comprise a set of 

instructions adapted to produce a suggestion re- 
garding the blocks to be selected. 

5 

22. A display device as claimed in any of claims 18 to 

21 , wherein each of the image data blocks is pro- 
vided with an identifier and the selection and fetch- 
ing of the blocks is based on said block identifiers. 

10 

23. A display device as claimed in any of claims 18 to 

22, wherein said compressed image data blocks are 
stored in an image data field of an image data stor- 
age unit and information associated with the selec- 

15 tion of image data blocks is stored in another field 
of the image data storage unit. 

24. A display device as claimed in any of claims 18 to 

23, wherein the image data is stored in a lossy for- 
20 mat. 

25. A display device as claimed in any of claims 18 to 

24, wherein the arrangement is such thatthechang- 
es in the image to be displayed do not affect the 

25 image data stored in the data unit. 

26. A display device as claimed in any of claims 18 to 

25, comprising a portable device. 

30 27. A display device as claimed in any of claims 18 to 

26, comprising a mobile station. 

28. A display device as claimed in any of claims 18 to 

27, comprising a digital camera. 
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